Sense distribution

For all the cases we have at least two homonyms, of which at least one is polysemous. The sense tags have codes described in a table with definitions at the beginning of each section, but the annotators also had the option of assigning a geen ‘none of the above’ tag, in which case they had to add an explanatory comment.

When setting up the annotation procedure, pilot batches of 40-50 concordance lines of each type were collected to estimate the frequency of the senses we expected (we did have to exclude candidates because some sense was not frequent enough). The annotation of the pilot sets was not extremely thorough and sometimes we modified the definition set afterwards, so it’s best to keep that in mind when reading the comparison between the expected distribution and what came out of the annotations. It’s worthy to consider that if my estimates over samples of 40-50 tokens match what we find in a bigger sample (and especially if it’s robust across batches) it’s quite encouraging. If (the skewness of) the sense distribution turns out to be a factor in the topology of the clouds, it’s useful to know that it can be estimated from such a small sample.

The Sense distribution subsection of each section compares then the estimated sense distribution based on the pilot concordances with the one found in each batch and the whole set of tokens. For each type a plot is shown with a row of dots per batch over a line and two more rows below the line representing the pilot-based estimate and the overall distribution. Circle dots represent tokens tagged with a given sense by the majority of the annotators (each sense has a color) and triangles represent either tokens primarily annotated with the geen ‘none of the above’ tag or tokens for which the annotators did not agree at all. The dots in the batch rows represent one token each and their transparency codes the mean confidence after standardizing its value by annotator and lemma.

Confusion matrix

For each type, two confusion matrices will be shown in the Confusion matrix subsection. In each of them, a row represents a majority sense (or no_agreement if there was no majority) and a column represents a sense tag. By hovering over the row names it’s possible to retrieve the definition, since the tags are not precisely transparent; columns are also grouped by homonym. Since the geen ‘none of the above’ tags can have different reasons, the annotators’ explanations were classified with the following tags:

• between, when the annotator reported doubt between two or more of the given senses;
• not_listed, when the explanations referred to a sense that was not contemplated in the list of senses (or not understood as such);
• unclear, when the explanations referred to either insufficient or unclear context (or simply difficulty to understand, such as “geen flauw idee”), and
• wrong_lemma, when they referred to an issue with lemmatization, part-of-speech tagging (including parts of proper nouns) or even spelling, so that the target didn’t actually correspond to what was meant to be annotated.

The first matrix shows raw annotation counts. Each cell tells the number of tokens with the majority sense of the row that were tagged with the sense of the column: The cell in the row of horde_1 and the column horde_2 will say how many tokens with the majority sense horde_1 received some horde_2 annotation. The totals indicate the number of tokens that were tagged with a given sense (for column totals). The first descriptions only focus on which senses are confused with each other. The caption also records the proportion of tokens with a certain majority sense or homonym that received the same tag from all annotators.

The second (“weighted”) matrix shows the mean of the mean confidences of the annotations. Suppose the row is horde_1 and the column is horde_2; to fill in such a cell, for each token with majority sense horde_1 the mean of the confidences of the horde_2 annotations is computed. Since the horde_2 is not the majority sense, there won’t be more than one annotation of the same token to average across; for the horde_1 column, each token would have two to four agreeing annotations, and their respective confidences would then be averaged to reach one mean confidence per token per sense. The final value of the cel is the mean, across all tokens of that cell, of those mean confidences. Here it is important to take into account that the annotators had to assign senses rather than homonyms: very often, the disagreement between sense annotations becomes agreement between homonym choices; I expect the same happens when confidence ratings are low in polysemous items.

Nephology

For each type, the Nephology subsection discusses the role of the parameters in the structure of the cloud of clouds (level 1 of the visualization) showing at least two color coded plots, and then compare sets of models (level 2). First, parameters that seem to have little to no effect in the variation between models are kept constant to compare the resulting selection; then other combinations that might provide different results are explored, and finally some combination of parameters that seems to provide “satisfying” models is kept constant look at the actual effect of the less important parameters. Normally, the strongest ones are those that select first order context features, while the second order parameters rarely make much of a difference.

The comparison between models normally takes the following steps:

1. examine the range of distances between the models through the distance matrix;
2. describe the general look of the clouds without color coding and how they change between MDS and t-SNE solutions;
3. color code with homonym and sense tags and describe the revealed structure.

The description includes the behaviour of outliers in MDS solutions, the separability of homonyms/senses in any kind of solutions, how many and how clear clusters show up in the different t-SNE solutions (if there is any that provides a particularly good representation, how robust it is across different perplexities) and how such structure relates to the parameters under comparison. For now, individual tokens are not examined but on an exceptional basis. A certain bias should be acknowledge: certain settings tend to be preferred (from theoretical reasons sometimes, but not always) and the findings in one type definitely affect how the following ones are understood. Hopefully, time and experience will provide the tools to revise these decisions with better criteria.

At the end of the subsection a next course of action is suggested, such as promising model(s) and which tokens seem interesting to look at. That is highlighted in a nice quote block at the end of each section.

To ease the descriptions, the parameters will be written in all caps and their values will follow them separated by colon. These are:

First order part-of-speech (FOC-POS)

Can take the value FOC-POS:nav, when only nouns, adjectives and verbs were selected as first order features, or FOC-POS:all, when there was no such restriction (still, some part-of-speech tags were ignored always, such as interjections).

The tendency is to default to FOC-POS:nav, since function words are probably less informative (kind of linguistically informed default).

First order window (FOC-WIN)

Can take the value FOC-WIN:5, when only features within a 5-5 window of the target were included, or FOC-WIN:10, when a 10-10 window was used.

The tendency is to default to FOC-WIN:10, to allow for more information; normally relying on other restrictions is enough to filter out the noise.

Positive pointwise mutual information as filter (PPMI)

Can take the value PPMI:weight when the second order vectors are weighted by the PPMI value between the first order feature they represent and the target type, PPMI:selection when only features with a positive PMI with the target type were included but the vectors were not weighted, and PPMI:no when no such filter was applied. Normally, the models with PPMI:selection are more similar to PPMI:no than to PPMI:weight and they are not considered in the initial comparisons.

The initial tendency was to default to PPMI:no, since a high PPMI value signals a feature as characteristic of the type rather than of groups of it (like a sense), but in the analyses described in this file it never performs as well as the alternatives.

Vector length (LENGTH)

Can take the values LENGTH:5000 and LENGTH:10000 when the 5000/10000 most frequent features were used as second order dimensions, or LENGTH:FOC when the same first order dimensions are used for the second order. That means that their number and frequency depends on the result of the first order restrictions for that particular sample of tokens. Normally, while this is not an extremely strong parameter, LENGTH:FOC can make a different against the other two, frequency based values.

The tendency is default to LENGTH:FOC because it should be better tailored to the specific context of the tokens in the cloud; it’s harder to compare clouds with different first order context words, but it does seem to perform better in most cases. Between frequency based values, I almost never look at LENGTH:10000, since it almost never seems to make much of a difference, but I probably should look into it before discarding it from future clouds. If both frequency based settings perform very similarly, smaller numbers should be preferred (hence the tendency to choose LENGTH:FOC as well, since it normally means fewer than 5000 dimensions).

Second order part-of-speech (SOC-POS)

Can take the values SOC-POS:nav or SOC-POS:all and refers to a filter on the second order dimensions. This never makes much of a difference.

The tendency is to default to SOC-POS:nav (De Pascale, 2019, pp. 62–63).

Second order window (SOC-WIN)

Can take the values SOC-WIN:4 or SOC-WIN:10 depending on whether the PPMI values for the second order vectors were computed based on a 4-4 or 10-10 window.

This parameter seems to group models for some types, but doesn’t really affect the structure of the clouds that much as far as I can see. The tendency is to default to SOC-WIN:4 (See De Pascale, 2019, pp. 62–63). Could it be that in the cases where it seems relevant, what actually happens is that all the other parameters are just too weak?

Eventually, it would be nice to reinstate sentence boundary as parameter (replacing for example SOC-POS). The difference between LENGTH:5000 and LENGTH:10000 also seems neglectable.